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This fnonocjroph hcis btu^n devolopi^j us a iJr.irt of tfw? dctivitips ol th(> Adv.inced Ins 
titutioncil Development Procjram (AIDP) Two year ColltMje Consortiurn. One of the eon 
sorliuni dctivllies is to develop publiccitions ihnt will be helf)fiil to consortium members 
nnd others in the hicjher education community. 

This monograph focuses on outcome me'asures in hiyher education. 1 fie [)ul)!ica 
tion has grown out of a workshop on tbis t0f)ic: held by the consortium ifi Washington, 
D.C. on March 17, 1976. 

The monograph consists of an article by Dr. Sidney S. Micek on the definition, col 
lection and use of outcome measures; an article by Dr. L. James Harvey on how outcome 
measures fit into a Planning, Management and Evaluation System; and two articles from 
colleges that are developing outcome measures systems. Dr. Enrique Solis, Jr. and Mr. 
Richard Drum present the conceptualization of a project at El Paso Community College 
and Dr. Charles J. Kinnison, of Kalamazoo Valley Community College, describes their 
project. 

The appendices contain some additional information on c tcome measures, includ- 
ing the measures tentatively defined by Kalamazoo Valley Community College. 



L. James Harvey, Ph.D. 
Contributing Editor 
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III. AN Af^PfiOACU TOOUTCOMl- MI.ASUfU.S 
OE l-INITION, COLLl-CTION, AND IJSC 



iNI KODIU I ION 

Tlu! ij(K)pl.il)i!itv i\nd r i»s[)t)iisiv(.MH-iS of coinnuinilv (:()ll(Mj(?s (o tin; ru.M.'ds of individ 
U(}ls diid s<jcicty litiv(? loru) hi?L'o tin; conc;(.'ro of cornnuinily coIIlmh? [jkiruKMS jnd dt'ci- 
siofriTijki.Ts. To(J.)V, liowL'Vt.T, tfu? [)r(?f.surus coinirnj from outsido tlui institLiti'>)n from 
ItMiiskUors, tiixf)Liyors, (Mii()!ovi-rs, f).)r(?nts, new apf)li(UHits jnd otIu.Ts as wrll as [iressuros 
l)t;itu) oxtTturl from witliin - from students, faculty iind .idrniiiistreitors ■ sucjyest tfuit 
iu)vv, more thiin t^ver l)(?foro. conirnunity (:ol!(?cj<?s ii(M?d to r(?exL}mirui tin.' results ond tMMie 
fits of tfi(Nr [)roor.i:ns from the f)ers[)(?ctive of tfie corn[)lex nnd (iyneimic issues confront 
in(] postsecorul.iry (?duCcition jnd to [)kui nccordincily . 

While i:ofnn;unity rolletje pkinr^ers and decisior^ makers are well awar(? of tfie issu(?s 
confronting tfieir instiLulions, they are beginniruj to recognize that sound directions can 
not be evolved by conc( Ur.^dng solely on historical cost data. Likewise, they are aware 
that the detailed analysis of demographic data and institutional re-sources does not pro 
vicie a final answer for makirv'i decisions and faying plans that will fielp the institution 
(?ffectively adapt and responn to the needs of students, employers, and other constituents 
at the local and state lev(?ls As a result, decision-makers are recognizing tliat for planning, 
management and evaluati^>fi be effective, these processes must be oriented to making 
the outcomes of communi:y college programs relevant to tf^e present and future ne(?ds 
of individuals and society. In snort, there has emerged an understanding of the necessi- 
ty for an "outcome-oriented" approach *o planning, management and evaluation that is 
based on information about the results and impacts of an institution's programs rather 
than on information based exdusivel/ on \Kha( i^ocs info such programs and how those 
programs are ope, 'ted. 

While most individuals concerned wiin the community college and postsecondary 
education in general, recognize the need and urgency for utilizing outcome information 
for purposes of program planning, management and evaluation, as well as to support and 
justify long-range plans and budgets, they are quick to point out the complexity of the 
problems associated with identifying and measuring the outcomes of postsecondary edu- 
cation and incorporating this information in the planning and b«.jdgeting processes. 

One major difficulty is that, traditionally, few explicit measures of program effec 
tiveness have been collected. Also, little has been done to show the links between re- 
sources and activities used and the attainment of desired outcomes, even when these out- 
comes can be quantified. In short, it has been muci easier to see whether a plan has been 
accomplished in terms of activity or resource measures (e.g., expenditures, student/faculty 
ratios, enrollment levels) than in terms of educational outcomes. 
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A sccnDil [Hohft'in ^>l(i'() I ili'tl r, i[),tl r\r\\ wImmi ihIdiiii ihoii jhoiit DUlroinri; ii. 
.iv.K l.ihli:, i( IS (lilluiill to iisr Miu;r tfic t tM ,1 u ) k ]i ii>s loi .ii i jl y i (H) .ind intrrpi (M ii hj (Iii'S(* 
(LiLi ,in' Imiiti'il i)[ .IP' not well uinltMstoixl. I rv,impit», c)iv('n dll ot thi? vjfuhlos tfi.it 
pi ) ten li.il I V .iMi't.l lu ul.ii odtcofix , it t » '\ 1 1 1 'i lu ' I v Jif lirult to (l(;ti;r nil ni) c.iusr <m(l 
r\\t)vA M'l.itioDsliip'.. A fur ;h(M i:of np li ' v 1 1 y M,"u)lts Immmusc m.iny pKHjuiitv; h.ivt» joint 
oil tiionit':.. f Of <.'\j(nplt' i\ voc.ition.il ircf'.nuMl pioqijin fti.iy (:ontfil>utc In :;tutltMit knowl 

.irn! skill lit: vtHopiiir'fil m .MMttiiMi t o j u oi |ui:i f )() v.ii m m is si.t vicrs t o niiMiil »i.'rs ot thr 
hdsint'ss V Oi ninui ii t y 

A inj|o[ (litiiciihy is ihjl most pLinnns .in I decision fnjkcfs siin|)ly f);iv(? .1 

hjfit tiinr U.insl.iPnM thcif in:;(itiition,il jinl ()foi]i.i(n mojIs nilo spocitic ohjfjct ivi?s stiUiMl 
ii) niiMSUr(il)u' outcofiu* trons iMditiondlly, i|o.il iicttiiu] is one of the first st(]f)S in tfi(» 
pi-iunint) f)ro(:(?ss; hovvrvin, oin-r tin: ()o^ils .)H.' st.tt('(f, loo oftun Ifujy rofiKiin ifi (jiMicrjl, 
fU)M {)[)(Miitiofuif itMiiis, jiiSf? ilif^ ijojls un:k tf^Hisf.ition ifito spucific, miMSu r;if)f»; cfo 
'vcnptions. [:>LHUU'rs .lOif in.iii.MjofS Imvi? tfoiif)lt» utili/ifi(j tfirni in suloctinc] tfio optirniil, 
or t'vt'n [)r()niisin(), coiksc:, of dction .ind ui t>v,ilii<ittn(| tf>*^ i nifjh'nionttHf proqriinis. 

F-'iiKilly, tlip list' i>f ' mlconic (fj(;i is fn thw:.if tiui liy tJir fiMf of [)(jU?ntijf niisusus. 
Oil'.* .ispL'ct of tfiis fisir i:; th.ii {\^^] (l.ir.i vvill not [loftr.iy j(t .H:<:uriito [)ictur(? of tfn: <!Ctuaf 
h«Mu?fits cl(»rivf!(l from tf)(' inslitutiofi jod its pro(Uiiins, A sticoiul. ancl p(}rhn[)S mor(? l)asic, 
coru:(jrn ts uncf^rtiiinty af^nut Uit? ultinuitt) findifu)^; iind tfit' mictions tfuit wiff l)e tciken l)y 
()t?rsons in [jositions of i:ontr(n oiitsidi' tfio instiuiiuuh Tf)is fnttor conc(M-n is bjsed on 
tfio f(Mr tfiat tf'io f'V(ilU()Hon [^rocfiss will not .ids-cjuatciy take into account tfiosc oiitcom(?s 
arid t)onofil5 tfiat art? non (luanlifiai)!^' and tfiosr infiuts aruj tio-Js tfiat are unique' to a 
qtvcMi institution or ()roc.]ran\ 

Tlif? roni[)lt;xity of idKMUi fy my, mtjasunny, and f.'valnatirnj t'u» outcomes of comrnuni 
ty coll(M)t?s and tfi(?ir ()ro()rams is of.wious and ovt?rvv!unminn; y^d, it is becorming increasiny 
ly clf,'ar tfaat we must !L»arn to deal witli ttieso prol)lems, given scarce resources, new de- 
mands frorn various consumers, and pressures for accountal)ility. The question tfiat ex 
ists is; "How do we [Mnjin to deal wirfi tiu^se pro[)lems?" 

Tfie overaif [)urpose of this monografjii is cJevotrnl to this question. The remainder 
of this paper is intended to share^ witli you the approach thdl has been taken at the National 
Centf^r for Higher Education Management Systems in trying to develop some useful tools 
and fJ^occdures tluit migfit help decisionmakers idcniify, obtain, and tisc tfie outcome 
inforniation they notid 'iox tlif.Mr plafinifKj, rnanagement and evaluation concerns. 

NC in MS AIMMU)A( M 

Although NCHEMS always recognized that botfi outcomf? information and cost in 
formation are nec^?ssary for effective planning and management in [)ostsecondary educa- 
tion, NCHEMS did not immediately attack both problems on tfie same scale. The Cen 
ter's initial efforts focused f^rimanlv on the develo[)ment of procedures for measuring, 
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,iiijly/iiu|, iind ruminiii lu ,it mil tiril i liiti' >i mI .iiui ()i ( .11 ii costs, 1 his st(.itt'(|v w.is not 
only ti fcsiih ()} 111!' M/r ol ill'' i)HthliMM jmi ihr Kiln'fi'nl i(";{niict' limit. iIhjiv* l)i)t jKo .1 
fcspofr.r to lfi»' Dt'cd'. .tiul pfrlcf ^[H■|'^ »)l Nclll! MTt'!; <:on'.>titm'nl:i. 

Ilowi^vcf, .IS I II r-;:;i )( rs loi iH:couf 1 1 jI )il 1 1 y ifu KMiii'U jikI iIu: posisccoiidary cducM 
tioi) ioniinunity Imhijii f(M;lii)i) tlx' fiiijnci.il pf(»s';iju>s of nsifu) costs .ind liinittHi rc 
souictis, NCfU MS f»M:oi)(ii/(:il tfi.it il no lofUHM could locus [uifiMiily ofi Ifii? dc?vi?lopnu:nt 
ot Ci»st loloiin.ition die pfol.ilciii ol dcvt)l()f)if uj outcome if ih m nut kjm luid to l)c .uldn.v. 
S(Mj ,ilso. As .1 fc'sult, the (lutcomcs of f\):;l:icM:onddiy Iducition Pjojiu:! wjs t?sl.il)lislu?d 
to develop tools .lud j xoceiluuis tor dt^sciihiiu) jnd iiiKily/idi) educjtioiuil outconuJS iind 
to iissist de(.;isiofi m.ikeis in uudei st jfidmM .iml usim) outcome' 'ufor mntiuu in their [)liiu 
uifUj t)(id miiiKKjemefi t decisiofis. 

At the hetjiiuiini) ot tiie Outcofiies fVoju'ci, tin ivuJjoi ol)jev:tive wcis to hocoiTie uiore 
f;imilcir with tlie outcomes medSuremeut jiid niKilysis p!ot)leni lu hi'.jher educitition und 
to dcUermiue how NChlEIVIS could did its cotistitcjttjnts in dtjalincj with the probleiTi Pui 
suiiu) tfns o[)jective, the staff he()cjn stiu.lyirnj wfidt luid heen done in this urou dnd examin 
int} the (jUf^'Stions decisionmakers were.' dskiiu] f t'lativt: to postsecondary tniucation ocJt 
(X)mes. Many discussions wcto held with prntitioners, researchers, and others intertjsted 
in tfie outcomes information [)ro[)lem, A national seminar was held in Washin(]ton, D.C. 
in 1970 for the purfK)se of high!if|htin(j the problems of identifying, measuring, and ana 
ly/ing tlie outcomes of f^ostsecondorv education. A nuriiher of authorities in the field 
were brou(]lit together to present position papers on the scjbject and the rescjits of the 
seminar were compiled and put.)li5hed (Lawrence, et ah, 1970). Another major activity 
during this early stages was the establishment of a Design Committee for the purfDOse of 
conceptualizing and stimulating thinking about what research, development, and imple- 
mentation efforts should l)e undertaken. 

As t[ie project progressed, this early objective gave rise to an overall conception of 
the outcomes area that could serve as an operational basis for future work. The long 
term Outcomes of Postsecondary Education Project currently underway at NCHEMS 
rests on the premise that, in order to collect and use information about the outcomes of 
postsecondan/ education, three related needs must be filled: 

1. The need to provide a comprehensive picture of the outcom.es of postsecondary 
education and to develop the capability to measure these outcomes; 

2. The need to provide a structure for organizing outcomes information as a prereq- 
uisite for the aniysis and communication of this information; and 

3. The need to develop analytic procedures to apply this information to tiie solution 
of particular planning and management problems. 

Mo)or comi^onents of the Outcomes Project are directed toward each of these nec'ds. 
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With irij.iM 1 In ///('./s/y/r.v I ; I ■ 'ilih'Jl M " i.il Mill, , >nu' . i h I !( M: - i wu i k i iii] U > ( I ) i ( li 'i i 
tify ini!;(Mls)i'i puslsrci hk l.ii v rilin:.thi»n mili ninr,. (.M <li'lin<' tMih nt Ihrsc milicilor. 
Ill .1 Sljiuj.ir( I I iMtUMi, {'A) ( level n| > ( w i >t -ei j( i f t Ii m jii)ntmi'i lijt j leLil ivr In iMch n I thi'se 
IfulKMlOls, .111,1 (.1) (||/,eiiiii). Mr Ihe 1 1 1 1 1 w 1 1 ■ ■ 1 1 1"; i .nn! [)fMihii I-. In O.m r.ioii iihlKris ill ihe 
pi )St:irr()n( 1 i(\ cilin jiion i i wn' n; m ii i v 

1 nM)i)i f.ictuc. .11!' l)rini| roMMili'i 1 III Hi- Ji 'vm ■ U i} u urn I at I lie tnll f.ifUM' '^t I 
va.'coi u iiil V nlui.Mliufi DUlmiin' inUic.ili w I ir.f, !ii"V iiii)''t lellriM J i )i|f ill loii lh<il in 
lof m.il u )ii nrci hi v.jf y roii'.K ItkiI il y .ht ( i !i I f ''h 'i 1 1 < Ici im« ui ni.il' fi s, !)rri 1 , U hums 
iif ("i niir;l jckf loaleilin? i li.ii ci hir .il i' )n.tf < >iM(\ mim' . n( ^ 1 1 f uvi'i .id ex ttMided per lod ot linic. 

In M'S()i)f ulifi'i ''u ifir' iit'tMl fo( ,11) Mn:.////////i: \f nu-f nfc , iNd II MS is, ( 1) ilevulofiifU) 
J <}(?n(Miil iMtiMjoi i-Mli< )ii 'ii.iicnic tui t;Ki:iM f\'ini) .iml iMii.ini.'iiu] ouleDnu.' [ihmsum.'S. (2) 
ujentifyifUj (irain(:ll\ di t Ici iMi 1 lypes ut etiin Mt tnnji uu Ir.iirncs, ('A) oiLMjur i/ iiu) Uic 
intMSurcs in arcor ' l.inct^ with ttiDSe disi inciiuns jimI (-1) [)f()VM)in() (uoctHlu rt?s wlu.'rnl)v 
niutisiires c.in Ijc jrf jyod wilhin lliesi; mj)nr (MtiMjor irs in j lotjic.il jnci useful tasliion, 
TIk? structurt' sliouUI nut only »i(:(:(niinu)tKjtc liic full , jivji: o\ ( ui tci wiu?s, l)Ut .ilsu indki! 
.)llc)Wijnc('s f(.)f till? v.if KHis uulcunn' diin^MisiufiS, surii ds l(Mi*j Icrin vltsus sl^nrl term jnd 
tfiL' niulti[)l(? levL'ls n{ fk't.iii (IrDfii milpyidujl ■.ocit.'ly) th.ii nuist he <iccomfnu(lr.ii(?ch 

in ^iddition to uf ij^iut/iiui intusuies t.e rj',- cufnnnmicMlion, lliis slructun? should 
[)rovKie d focjl [)oint f(jr e, )n(:ef)t u.il ond thei)nMn:.»! discusMons surroufidincj llu; cjues 
tiou, "Just whcii arr tlie oulconies of [losisi^cofuhiry educriliun ' The cjev(?lM[Tnieni of a 
coinfKehensivo oulconies structure ref|uir"s tli(? resolution of ifu? very difficult (|uestion 
of wlKit sliould l)e^ considererj an oijteonu.' uf postseeondary eciucation and wiial should 
not. The sirucUjre will serve cjIso to identify t:dltMjui u,'^ of outeonn?s for whicli ineasures 
or inciicators are f^ol ytd irlenii f iecj. Used in tfiis way, a lornial st uclure can not only 
f)rGvi(ie a mechafiisin to assist in tfie coordination ot research and development efforts, 
but also senye as ..i device) to hnjhlKjht tfiose outconies for whicfi quantitative n^iasures 
are not availal^le, tl)us reinforcinq the' need te^ d(Ml with such outcomes cjual i tatively. 

Inasmuch as tlie ultimate vyl^jective is irn^:»roved [:)lanning, rnana(iement, and evalua- 
tion, NCHEMS is f)lacinq [)articular erTi[:)hasis on tfie identification and development of 
relevant analytic rapahilidi's that will foster the interf)retation and use of outcome data. 
In particular, attention is heirn] focuseri on develo;:)iria capabilities for investigating rela- 
tionships among inf)ut or \)roo ss variables and resulting outc:onies. It is anticipated that 
users will Ijenefit not only trom tne actual findings of these analyses in case study form, 
injt also from ilie diss'Mnination of prototype analytir: [procedures and tecfmiques ap- 
plica[:)le in undertaking enalyse^s of f)articu!ar kinds of ()roi)lerns. 

In carrying out thf^se tasks. inJCHEMS is relying iMMvily (jn a strategy of ()roducing 
f)roducts early m the df^velopnuMU cycle and then concentrating on niaking iniprovemen ts 
wherever po5sil)le. In kee|)ing witii tfie techneguf.*, ti ()relimiruiry set of outcome measures, 
a [jrcHiminary outcorTies ckissif ication structurf.'. -ind an initial se't of outcomes flata collec- 
tion and analytic t)rocedure-s anu' guiflelifies have [)een fl{.'ve'!o[)f^d and are l)eing fietrj testenj 
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Si j} )',{-( j urn I rM( If I', Will \)r t lifcitcd ti ivv.ii 1 1 m in i ".m vr I , iqf, ( > I c x 1 1. it im ni i . im I f < I men h-i 1 1 
lit thr I K ( I', m iM( h t >f tlu' tliirr ,ihm', 

\\\r <lt'V('ln[MniMii '(in»,M:h i'. jIm) thji.ictri i/cil I ly ihf ,ii;tiv(' nivi^lviMin m 1 1 ot tlu- 
I in In )t Ml usi-n, 1)1 (In- icMilt'. vil (Mcli fn.i|( )i Nta(H', M!i Ims i:t)n:.iMi'nlly t,i 

\aw(h1 .1 i.lylr ol ( m -il Ui.it '.dIiciIs input Imin piodiict iiscis duf iiM \ Ur ( it'V(.'ltM)ii irnl 
piociv;-.. I Ir.ttMHMlly, tins iovim^imdimU hriM) olM.ittUHl thioiK]!) l.isk ttMi'C.. jdvisoiy 
i|r()ij[):i, tu'lii it'vii.'ws Jini [)iU)l l<v»ts hi llu' Oiitcofin-; rfoiccl^ llu'Sf involvrintMi I itu-ch 
ufusins Im?im() jiitjnuMitt.'d l>y surv<!ys Jfid Dlhcf df^vicrs di.it .illow i.t.itt to .it:((ijiii* 
lir.i-r iiifiut ifi <i fnof'i' :;yst(Mnift (ifid sfitTitu: vv-iy 

\ roSrSI ( ()M) AKV I DIK Al ION 

Oi; I ( O.MI S ( I ASSII k A I ION SI KIU I IIKI 

Ofie of striituqii.'S l)uin(i (Mii[)loyt?d in tin' jttmnpt to .iccornplisd \\m: ol)j(H:tiv(is 
oudined tjl)ovo is tho duvolof)mi?nl of a cldSsiluMtion stri)Ctur») that can simvi' not only <is 
an or(]ani/in(j framework for thu i:oni[)rolu?nsive set of postsocondary education outcrorncs, 
hut also as an operational focal [)oint for consi(Jeration of tfie cjuestion, "Just what arc 
tiie outcomes of postsecondary educationf^" An inihal vc'rsion of tfiis slructun^ already 
fias been dfvelofXKi in tfie form of an h^vontnry of f^njhar Education Outco/fU' Voriahlcs 
and Mo^miros (Micek and Wallhaus, 1973). The inventory is l)ased on the f)rer7iise that 
aithouofi improving the ability to acquire information i\boui outcomes is critical, the 
mere availability of this information is not the ultimate objective. The objective is tfie 
delineation of decision-relevant data and tfic improvennent that can result from the use 
of ttiese data. A primary use of such data is in tfie evaluation of tfie effectiveness and 
efficiency of tfie alternatives available to the decision maker. Sucfi use invarial)ly involv(js 
comparisons of botfi costs and benefits, that is, comparisons amonr; ()rograms or insti- 
tutions, or between tfie same [)ro(}ram or institution at different points in time. 

To make such comparisons, one must communicate outcome information among 
different institutions, different programs, and different kinds of decision-makers, Tho 
requirement for a communication capability, in turn, creates a requirement for a com- 
mon language permitting this communication. Tfiis need for a common language to pro- 
mote the use of outcome information has several major implications. First, it means the 
measures of educational outcomes must be defined in some standardized way. Just as 
financial cfata must be defined and aggregated in uniform ways for cost information to 
be comparable and useful for communication without misinterpretation, so must data 
about outcomes be defined in uniform ways in order to be comparable. 

Second, the need to communicate outcome information creates a requirement for 
some uniform scheme of organizing and categorizing the individual pieces of informa- 
tion. Without some framework or structuring device, communication becomes exceedingly 
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1 P Student (jfowth .ind IVivtHopniirnt 

1 1 .() K Ml jwiiM !()(.' .irid likills Oi.'vrlopincnt 
1.1 1.00 KnuwItxJyt; D(n't.'lo|)mt'n; 
M.2.00 Skills DL'vtjIopnipni 

1 . 1 .3.00 Knowtf.'dijo .ifid AtliTudcs. V.iltH"., ,ir>':! {iriicfs 

1 ? {) Soci.il D('vt!lof)fnf'nt 
1 2.1.00 Socidl Skills 

1.2.2.00 Soci.il AttituH*'^ V.tlur-., ,ina Hrlirt^ 

1.3 0 Porson.i! D(^vtMopniont 

1.3.1.00 Stu'l.'fc Mr.ilTh 

1 .3.2.00 Persor^al Attitjflos, V.ilut?'. .ind Bi'lu'l'^ 

1 .4,0 Cjr<jer D(.'v**lopfn'.Mn 

1 4 1.00 Carn^T PrepiHatiun 

1.4.2.00 Cdroor Attitudes, Vtiliit^s, ciiid Rcl.rfs 

2.0 DevelofmiL^nt of Nl'w KnowUKifj^? .ind Art foiiT^s 

3.0 Community Devudopmenr .'jrKl Service 
3.1.0 Commur^ity Development 
3.2.0 Community Service 
3.3.0 Longer-Term Community Effects 
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Within each area of the Inventory, general outcome variables have been defined, to- 
gether with specific potential measures of those variables. Thus, such variables as develop- 
ment of general knowledge, developm * specialized knowledge, and critical thinking 
and reasoning skills have been identifieo ch the knowledge and skills development area. 
Measures that might be used to assess these variables include student scores on various 
standardized tests, numbers of graduates accepting employment in their major field of 
study as a percentage to total graduates in that field, or average student and/or former 
student scores measuring their degree of satisfaction with their ability to apply what 
they k.now. A complete discussion of the development of the Inventory and its possible 
use is given in the document. An Introduction to the Identification and Use of Higher 
Education Outcome Information (Micek and Wallhous, 1973). 

One additional issue that needs to be addressed with respect to the Inventory is the 
question of its use. Several potential uses can be identified. First, the Inventory should 
greatly facilitate communication between decision makers in all sectors and levels of post- 
secondary education. That is, it should serve as a "common currency" for exchanging 
ideas and information about the results of postsecondary education. It should provide 
a common language that will assist all who are interested in improving postsecondary ed- 
ucation to understand specifically what one means when reference is made to a particular 
outcome variable. 

The second primary use of the Inventory is to facilitate the planning process in post- 
secondary education. Without a clear understanding of the direction one wishes to go, 
that is, without knowing the objectives one wishes to achieve, it is nearly impossible for 
the planner or manager to determine which alternative course of action will provide the 
best investment return or the best benefits. The Inventory should be helpful specifically 
as a device for translating broad goals into clearly defined and operationally specific ob- 
jective statemeirs. For example, a planner with a set of broad goals to guide him will be 
able to pick and uhoose those outcomes he wants to achieve from the full menu of out- 
comes. One he has developed his specific list of desirable outcomes, he then can rank 
them and compare the expected outcomes of the various program alternatives. This use 
of the I nventory's cateogires will serve also to highlight those goal areas for which quan- 
titative measures are not available and thus reinforce the need to deal with such outcomes 
qualitatively. The third primary use of the Inventory will be to aid in evaluation. Often, 
when one wishes to compare the actual outcomes of a given program to the desired out- 
comes, the process is thwarted because the desired outcomes have not been spelled out 
clearly. It is expected that by identifying the specific outcomes from the lnvento>-y the 
evaluator will know precisely what is to be achieved, and subsequently, he will be able 
to make an accurate comparison of the actual and intended outcomes. Such accurate 
information then can be fed back for decision making about the addition, deletion, or 
modification of future programmatic efforts. 

Finally, since the eventual objective is to develop a structure encompassing the "full- 
range of educational outcomes," further developmental activity will serve as a focal point 
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for the conceptual and theoretical discussions surrounding the question, "Just what are 
the outcomes of postsecondary education?" The development of the Outcomes Struc- 
ture requires che resolution of the very difficult question of what should be excluded. 
To the extent that answers to these and other basic questions will require further anal- 
ysis, this effort will identify areas in which future work should be done. 



THE OUTCOME MEASURES 

IDENTIFICATION STUDY 

Another key component of the NCHEMS Outcomes of Postsecondary Education 
Project is the Outcome Measures Identification Study (OMIS) procedures. The purpose 
of these procedures is twofold. First, they are intended to identify those outcome mea- 
sures or indicators perceived as most needed by different decision-makers. This process 
also should reveal differences and similarities within and among various decision-making 
groups with respect to outcomes information needs. Second, implementation of the 
OMIS procedures should identify those outcome measures that should receive top pri- 
ority with respect to the identification and development of needed outcome data ac- 
quisition procedures. 

NCHEMS has employed the Outcome Measures Identification Study procedures to 
survey both institutional and state-level decision makers. The institutional group surveyed 
by NCHEMS consisted of the president and top-level administrators for academic plan- 
ning, student affairs, and budget and finance in a sample of community colleges, public 
and private four-year colleges, and public and private universities. The state-level group 
included state directors of higher education and community/junior college boards (or 
their equivalent), state legislators, legislative analysts, state budget officers, and governors. 

The survey questionnaire used in the OMIS is based on the outcome measures con- 
tained in the Outcomes Inventory previously described, and is designed to: (1) identify 
the outcome information areas decision makers feel are most important (see Table 2), 
and (2) identify the extent to which decision makers "need to know" and "have access 
to" specific outcome measures. For example, in Outcome Area D, which contains a set 
of outcome measures related to Student Occupational Career Development, a decision 
maker had the opportunity of indicating his perceived need (or lack of it) for each of 13 
specific measures (see Table 3). 

Various descriptive analyses of the data obtained from the OMIS surveys have been 
conducted. The results of these aniyses are reported in The Higher Education Outcome 
Measures Identification Study: A Descriptive Summary (Micek and Arney, 1974) and 
in The State-Level Outcome Measures Identification Study (Micek and Oberbeck, forth- 
coming). Table 4 lists 20 outcome measures that 60 percent or more of the respondents 
in one or more of the groups in the two NCHEMS surveys indicated a "need to know." 

14 
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Table 2 



Outcome Information Areas Contained in NCHEMS 
Outcome Measures Identification Study 



A. Student Knowledge and Skills Developnnent 

B. Student Educational Career Developnnent 

C. Student Education Satisfaction 

D. Student Occupational Career Developnnent 

E. Student Personal Development 

F. Student Social/Cultural Developnnent 

G. Community Educational Development 

H. C'-mmunity Service 

I. Community I mpact 

J. Development of New Knowledge and Art 





In general, the survey questionnaire and procedures used in the Outcome Measures 
Identification Study have worked quite well. The results of the surveys not only have 
helped in iden' :*ving the outconne measures that are important to college and state-level 
officials, but a.i ave confirmed that different types of decision makers perceive a need 
for different kinds of outcome information. 

In a practical sense, it should be noted that certain aspects of the OMIS can be used 
by institutions for identifying the outcome measures important for internal program 
evaluation and external reporting. For example, one institution that participated in the 
original pilot test of the OMIS procedures has used the responses of its administrators to 
identify those outcome measures for which they should begin developing a data base. 
Based on the list of measures identified in the results of this particular institution's sur- 
vey data, work now is underway toward developing procedures for obtaining the necessary 
outcome data as well as the other kinds of input and process data necessary for interpret- 
ing the outcome information. This use of the procedures and results of the Outcome 
Measures Identification Study is encouraging since it suggests that institutions on their 
own can begin implementing the OMIS procedures. 
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Table 3 

S.tudent Occupational Career Development Outcome Measures 



1. Number and percentage ur fo.mer students {graduates and nongraduates) sur- 
veyed who were employed within a certain time period after leaving the insti- 
tution. 

2. Number and percentage of former students (graduates and nongrdduates) sur- 
veyed who received the job of their first choice. 

3. Average first salary of former students. 

4. Distribution of former students (graduates and nongraduates) across income 
categories within a certain time period after leaving the institution. 

5. Former students (graduates and nongraduates) scores on a scala measuring 
their degree of satisfaction with their job performance. 

6. Number of professional occupation awards and citations received per former 
student surveyed. 

7. Number and percentage of former students surveyed who are in management 
positions within a certain time period after leaving the institution. 

8. Number of voluntary/involuntary changes in employment within a given time 
period per former student surveved. 

9. Number of voluntary/involuntary changes in career field within a given time 
period per former student surveyed. 

10. .Average first salary expectations of students. 

1 1. Numbfir and percentage of students who are aspiring to a particular type 
occupational career. 

12. Number and percentage of students and/or former students surveyed who 
are seeking certain levels of employment. 

13. Number and percentage of former students (graduates and nongraduates) 
surveyed accepting employment in their .major field of study. 
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Table 4 



Outcome Measures Endorsed by 60 Percent of 
One or More "OMIS" Survey Groups as "Need to Know" 



1 . Number of students passing certification or licensing exames (e.g., bar exam, CPA, LPN) 
on first attempt ab a percentage of all students taking the exam. 

2. Student scores on tests that indicate their ability to read, write, speak, and/or listen. 

3. Number and percentage of students surveyed who have participated in activities that en- 
hance their communication skills (e.g., debate, encouter groups, etc.). 

4. Number and percentage of students surveyed identifying a certain degree diploma or cer- 
tificate as the highest degree planned. 

5. Number and percentage of students surveyed who are taking noncredit, independent study, 
or special courses. 

6. Number of students receiving a degree, diploma, or certificate within a certain time period. 

7. Average amount of time it takes a student to earn a degree, diploma, or certificate. 

8. Number of students graduating from the institution after a certain period of time as a per- 
ctntage of their entering class. 

9. Number and perf:entage of graduates for the year who transferred from another school. 

10. Number and percentage of students leaving the institution prior to receiving c degree, 
diploma, or certificate during a particular academic term or year. 

1 1 . Student scores on a scale measuring their degree of satisfaction with thei; ^Jrogress in 
achieving their educational career godis. 

12. Student scores on a scale measuring their degree of satisfaction with their progress in 
achieving thei - occupational career goals. 

13. Number and percentage of former students (graduates or nongraduates) surveyed who 
were employed within a certain time period after leaving the institution, 

14. Number and percentage of students surveyed who are aspiring to a particular type of oc- 
cupational career. 

15. Number and percentage of former students (graduates and nongraduates) surveyed ac- 
cepting employment in their major field of study. 

16. NumbD^ cf nonmatriculating participants enrolled in instructional programs as a percen- 
tage of the total number of persons in those programs. 

17. Number and percentage of graduates of a particular graduating class who are employed in- 
state versus out-of-state. 

18. Community attitudes toward the institution (e.g., attitudes toward the institution's con- 
tribution to community social/cultural activities) and the institution's impact on the 
amount of crime in the community. 

19. Number of proposals funded for certain purposes (e.g., research versus training) by level 
of funding as a percentage of all proposals submitted. 

20. Total dollar amount of gifts and/or grants received for certain purposes (e.g., research 
versus training) as a percentage of total budget within a certain time period. 
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THE OUTCOMES MEASURES 

AND PROCEDURES MANUAL 



A final major effort of the Outcome Project has been the development of the Out- 
come Measures and Procedures Manual (Micek, Service, and Lee, 1975). This manual 
has been designed to serve as a flexible and practical guide for helping decision-makers 
deal with the problems of obtaining and analyzing a wide range of needed outcome in- 
formation. It does this by presenting an array of alternative procedures that can be used 
by institutional researchers, planners, and evaluators to obtain local data for a select 
number of institutional and program outcome measures. 

Selection of the outcome measures for inclusion in the manual and development 
of the procedures for obtaining those measures were based on five major criteria: 

1. The manual should, to the extent possible, bridge the full range of postsecondary 
education outcome measures. Measures of the outcomes of the instruction, research, 
community service, and institutional support programs associated with various 
types of postsecondary institutions should be considered for inclusion in the manual. 

2. The outcome measures entertained for inclusion in the manual should reflect the 
fact that postsecondary education outcomes occur over an extended time period. 
Therefore, the manual should consider measures of both short-term and long-term 
outcomes. 

3. The outcome measures and their corresponding data acquisition procedures should 
reflect a recognition that the information needed by different decision-makers 
varies considerably as to when it is needed and at what level of detail. 

4. The initial version of the manual should be primarily a compilation of the current 
state-of-the-art capability with respect to feasibility of outcome data collection. 
Future versions will incorporate improved and newly developed outcome data 
acquisition procedures. 

5. The manual should be a flexible and adaptable tool from which users can pick and 
choose the most appropriate procedures for acquiring data related to the outcome 
measures they need. 

The outcome measures and their associated data acquisition procedures presented 
in thj manual have been organized into three major sections: 

■ Student Growth and Development Measures and Procedures; 

■ New Knowledge and Art Forms Measures and Procedures; and 

■ Community Impact. 
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Table 5 cn the next page shows the structure used to organize the outcome measures 
and data acquisition procedures contained in the manual. 

For each outcome measure contained in the manual, a one-page abstract is presented 
that provides: 

1. The name of the outcome measure. 

2. The number used to categorize the measure. 

3. A definition of the measure. 

4. The data sources from which the data needed for the measure can be obtained. 

5. A listing of the general type of procedures recommended for obtaining the measure. 

6. Comments that may be useful in understanding the use of the measure and its ac- 
quisition procedures. 

Following the one-page abstract for each measure, the data acquisition procedures 
suggested for that measure are presented. In some instances, alternative procedures have 
been presented to give the user as much flexibility as possible in acquiring the data neces- 
sary for the measure. For instance, procedures may differ with respect to data collection 
mechanisms (institutional records versus questionnaire surveys) or data sources (surveys 
of existing students, former students, or administrative staff). A list of the outcome 
measures contained in the manual and an exempi jr data acquisition procedure is pre- 
sented in Appendix A, 

FINAL COMMENTS 

Clearly, pressures from both outside and inside the institution are making it para- 
mount that community colleges improve their ability to document and articulate the 
outputs and impacts of their programs. How to proceed in improving this capability is 
a difficult question in view of the inherent problems associated with identifying, measur- 
ing, and evaluating educational outcomes. This paper has summarized an approach the 
National Center for Higher Education Management Systems has taken in dealing with 
these problems and has reviewed three major NCHEMS products designed to help deci- 
sion-makers accomplish three tasks: 

1. Identify and classify the range of possible outcomes of postsecondary education; 

2. Determine who needs what kind of outcome information for this particular deci- 
sion-making responsibility; and 

3. Develop and implement procedures for obtaining needed outcome information. 
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Table 5 

Outcome Measures and Procedures Categories 



Student Growth and Development 

A. Student Knowledge and Skills Developnnent 

• Measures and procedures concerning student understanding, competencies, and attitudes relative to bodies of 
facts and principles and use of their intellectual and physical abilities. 

B. Student Educational Career Development 

• Measures and procedures concerning student attitudes and success relative to ceitain academic pursuits, e.g., 
student educational degree aspirations and attainments. 

C. Student Educational Satisfaction 

• Measures and procedures concerning the satisfaction of students about the knowledge and skills they have 
acquired and their progress toward their educational and occupational career objectives. 

D. Student Occupational Career Development 

• Measures and procedures concerning student attitudes and success relative to certain occupational goals and 
their job performance. 

E. Student Personal Development 

• Information about changes in students conerning the growth and maintenance of their personal life, e.g., their 
ability to adapt to new situations, their self-concept. 

F. Student Social/Cultural Development 

• Information about student abilities and attitudes in dealing with people and their interest in cultural activities. 

New Knowledge and Art Forms 

G. Development of New Knowledge 

• Measures and procedures concerning forms of new knowledQe developed, applied, and reorganized by an insti- 
tution's progranns and its faculty, staff, and students (current and former). 

H. Development of Art Forms 

• Measures and procedures concerning forms of art, e.g., n musical score, a play, a sculpture, created by an insti- 
tution's progranns and its faculty, staff, and students (current and former). 

Community impact 

I. Community Impact: Education 

• Measures arvd procedures concerning the attitudes and success of non-degree/diploma/certificate participants 
relative to their acquisition of knowledge and skills, personal ar>d social development, and occupational career 
goals and performance. 

J. Community Impact: Sen/ice 

• Measures and procedures concerning the impact of the opportunities and services provided by the institution 
and received by the community, e.g., agricultural extension services, cultural and recreational opportunities. 

K. Community Impact: Economic 

• Measures and procedures concerning the impact of an institution's programs and its faculty, staff, and studerti: 
(current and former) on the financial health and manpower supply of the community (local, state or national). 
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IV. PME AND OUTCOME MEASURES 
Dr. L. James Harvey 



The purpose of this section is to clarify the relationships between Planning, Manage- 
ment, and Evaluation Systems (PME) and Outcome Measures (OCM). In addition, some 
comments will be made regarding the linkage of Outcome Measures with Management 
Information Systems (MIS). 

First, let us focus on the definitions of OCM and PMF. 

PME. The process of establishing the college mission, goals, and the derivative goals 
and objectives, of coordinating the affairs of the college to achieve tlie goals and objec- 
tives, and determining how efficiently and effectively the planned achievements were 
realized. 

OCA/. A quantifiable measurement of the results or impact of an educational insti- 
tution or one of its programs. Outcome measures are designed to assist the college in 
identifying and collecting information needed to determine institutional effectiveness. 

If an inbiitution has an effective planning system, and if they are planning in terms 
of the definition above, then they are setting institutional objectives. These objectives 
stem from broader, usually non-quantifiable, institutional goals. The objectives, however, 
are by definition quantifiable and measurable. These institutional objectives and the 
measures they contain form the basis for outcome measures and for an outcome meas- 
urement system for an institution. The OCM becomes the heart of the institution's 
evaluation system since they focus the college's attention on the essence of what it is 
attempting to accomplish. The management process is between the planning and evalua- 
tion processes and focuses on carrying out the activities needed to accomplish the ob- 
jectives. 

Figure 1 is presented to assist the reader in further clarifying these relationships. 

It is assumed that in its objectives, an institution will focus on who it wishes to 
serve, how it wishes to serve them, and what impact it hopes to have on those it serves. 
To the extent that these matters are clearly stated in quantifiable objectives, the out- 
come measures are the quantifiable elements in the objectives. 

The objectives usually focus on desired outcomes or the expected outcomes. The 
final measurement determines what the actual outcomes are. These definitions of dif- 
ferent types of outcome measures are helpful in talking about OCM, but they are all 
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PME COMPONENTS 
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EFFECTIVENESS 



Figure 1 



included when an institution sets and measures objectives. There is, however, Ofie other 
category of OCM which needs to be mentioned because it is, by definition, not part of 
the goal and objective setting process. This is the tacit outcome measure. 

Tacit outcomes are those that the institution has on a person or community which 
it does not intend or that it is unaware of. For example, it may not be an objective of 
the college, but one of its impacts on an area is that it keeps a large number of students 
out of the full-time job market. In addition, a college may have a significant impact on 
a student's desires to learn which may not be in an institutional objective. Some objec- 
tives may be tacit when they are discovered, but be converted into desired or expected 
outcomes as they become the focus of planned institutional effort through the estab- 
lishment of objectives. 

Figure 2 represents an attempt to diagram a PME model that develops outcomes 
and outcome measures. This is only one of a number of models that could be developed 
and is presented to show the relationship between PME-OCM and a MIS. 
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Figure 2 



A fuMy developed management information system (MIS) would include the out- 
come measures. These measures should become an intricate part of this information 
system and a key factor in monitoring the progress and impact of the institution. In an 
ongoing MIS, the measures should be routinely gathered, formulated in meaningful re- 
ports and monitored regularly by the college trustees and executive managers. 

One last point needs to be dealt with. If a college does not set objectives in quan- 
tifiable items and does not have a modern PME system, does it still have outcomes and 
can it still have an OCM system? The answer is yes. Obviously, the college is having 
impacts and accomplishing results, whatever they may be. The extent to which these 
impacts and results can be identified and measured is the extent to which the college 
can develop an OCMS. The college can borrow an OCM system from someone else or 
copy the good work done by NCHEMS. This obviously is not the best way to do it, 
however, and is a little bit akin to starting a system by looking at the answers (mea- 
sures) before the questions (objectives) are defined or asked. In the judgment of the 
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author, an institution using the latter approach runs the risk of having more unidentified 
measures (tacit OCM) than identified measures, A system developed in this fashion would 
also be less valuable for management purposes. 

Whether a college begins its OCMS by focusing on its objectives or whether it begins 
by looking at other systems and available measures, these two elements will continue to 
work back and forth as the system becomes more sophisticated and integrated. For ex- 
ample, as the college sets new objectives, new measures will be added to the OCMS. On 
the other hand, the data developed in the OCMS may lead to the discovery of new indices 
or measures which in turn may become the basis for setting new institutional objectives. 

In summary, an OCMS is an intricate part of a complete PME system. The outcome 
measures form the backbone of the "E" (evaluation) in PME. They also serve as a major 
focus of the institutional objectives which are essential to the "P" (planning) and "M" 
(management) sections of PME. 
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V. OUTCOMES MEASUREMENT AT £L PASO 
COMMUNITY COLLEGE 

Dr. Enrique Solis, Jr. 
and 

Mr. Richard Drum 



This article should more aptly be titled "Why El Pp.so Community College is Under- 
taking a Program of Outcomes Measurement." A corollary to this would be "What do we 
expect from such a program?" This paper will address the plans and reasons for imple- 
menting an outcomes measurement project at the college. Implementation will proceed 
with the assistance of the National Center for Higher Education Management System 
(NCHEMS) as part of the College's Management Improvement Activity under its Ad- 
vanced Institutional Development Program (AIDP). 

The College is a unique institution in many respects. It was opened in the fall of 
1971 by Dr. Alfredo de los Santos, Jr., with an initial enrollment of 901 students. This 
initial enrollment mushroomed to a total of more tlian 1 2,000 students during the spring 
of 1975. 

Throughout this period the school has operated out of leased army facilities. In 1975, 
the College purchased and remodeled a facility which had served at different times as a 
high school and as administrative offices for a local school district. The building became 
the El Paso Community College, Rio Grande Campus, and presently houses all of the in- 
stitution's allied health programs. 

Also in 1975, a contested decision on a 1S74 bond election was decided in favor of 
the college district and plans are under way for the construction of two new campuses. 
When all the plans are finalized and realized. El Paso Community College will be a multi- 
campus organization with a central administration. It will be better able to^erve the com- 
munity, but will also have the multiplicity of problems associated v^'ith its tentacular struc- 
ture. Proper planning for efficient operation must begin now. 

The College has moved very rapidly in other areas. The faculty and the administra- 
tion have been very active in curriculum development and individualized instruction ef- 
forts. Since the school population is approximately 57 percent Spanish surnamed, col- 
lege personnel also have concentrated heavily on one very important thrust of the insti- 
tution - bilingual education. In addition, a system of Management by Objectives (MBO) 
has been implemented which includes faculty participation through faculty personal plans 
of action. 

In short, at El Paso Community College, we have an institution that has had over a 
ten-fold increase in enrollment in less than five years. It is on its way to becoming a muiti- 
campus organization, has progressed rapidly in individualized instruction, is moving to 
implement bilingual education, and has a working system of MBO. 



The MBO system at El Paso Comnnunity College focuses on managing the institu- 
tion through sets of integrated objectives with measurement indicators. This system has 
sepyed as the framework for the College's planning, management, and evaluation mecha- 
nism. There is, however, a weak link in the system which is crucial to its proper operation. 
That shortcoming is evaluation. Our methodologies and data collection schemes are neither 
systematic nor centralized. 

An exception exists. Our curriculum development efforts are subjected to fairly 
stringent evaluation procedures. Formative assessment of materials development and in- 
dividualization of particular courses is in operation from the proposal to the implen-en- 
tation phases. Learning outcomes, however, are not rigorously assessed. 

Another area that needs strengthening is the instituiicn's methods for assessing the 
impact of the College in the community. In the past, El Paso Community College has 
conducted: 

■ Employment needs surveys; 

■ Vocational student follow-up studies; 

■ Attrition studies; 

■ Some specific community impact studies; and 

■» Community service and continuing education needs. 

These and otiier studies, however, have been somewhat disjointed. They have lacked 
that thread of coordination that turns a series of information gathering schemes and de- 
vices into effective management tools. In this respect, our growth has outdistanced our 
capabilities in this area. The danger exists that if assessment methods are left unimproved, 
eventually the institution will manage the managers instead of the managers managing the 
institution. We feel that a program of integrated outcomes measurennent will provide the 
needed integrating thread. 

El Paso Community College's planned involvement in the NCHEMS program of out- 
comes measurement includes: 

■ identification of outcome information needed by key decision-makers 
on a recurring basis; 

» Determination of the uses different decision-makers will make of their 
desired ourrc.,',e information; 

■ Development of plans and procedures for obtaining needed outcome in- 
formation on a continuing basis; 
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Training staff in the application of syste: )atic procedures and instruments 
in specific job areas; 

Testing adequacy of the procedures for the various outcomes data collec- 
tion efforts: 

Modification of the outcome information plans, instruments, analysis pro- 
cedures, and reporting procedures based on evaluative feedback about the 
above activities: and 

Implementation of outcome information collection instruments on a con- 
tinuing basis and utilization of the information obtained for internal deci- 
sion-making and external reporting. 



EXPECTED OUTCOME OF 

OUTCOMES MEASUREMENT 

El Paso Community College is at a., advantageous point fronn which its staff and 
faculty personnel can progress to efficient management. The institution is very young, 
yet has made tremendous progress in a variety of areas. This combination of character- 
istics affords EPCC the flexibility to implement complementary management tools which 
provide for assessment of the impact it has had through its multi-varied innovations and 
programs. Participation in an outcomes measurement program should enable EPCC to 
identify, collect, and analyze that information that will provide better direction to de- 
cision-makers for more effective management of the institution. This improved base of 
information for decision-making is, in a nutshell, EPCC's expected outcome. Spin-off 
benefits will include; 

B Institution-wide common denominators as indicators of impact and ef- 
fectiveness; 

Staff trained in data identification, collection, and analysis of informa- 
tion: 

Administrators attuned to the decision-making process based in part on 
objective and scientifically processed information; 

o Staff skilled in the translation of goals into 'antifiable objectives; and 

Precise articulation of problem areas, i.e., a clearer distinction between 
actual problems and symptoms. 

It should be noted that a program of outcomes measurement is not viewed as a 
cure-all for any and all of El Paso Community College's management ills. It is only one 
tool for effective management, albeit a very kev one. 



- 24 - 



The program will be coordinated by the Office of Institutional Research. The Of- 
fice has been made possible by a Title 1 1 1 , Al DP grant, and has as one of its objectives ^ 
the assessment of college impact for feedback to decision-makers through a program o 
outcomes measurement. The combination of management activities of the AIDP grant 
should result in a more effective management system for the institution. Outcomes mea- 
surement is expected to play a key role in that management sytem. 
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VI. DEVELOPMENT OF THE OUTCOME MEASURES PROJECT 
AT KALAMAZOO VALLEY COMMUNITY COLLEGE 



Dr. Charles J. Kinnison 



A "management sytems" project now in process at Kalamazoo Valley Community 
College (KVCC) is designed to improve the College's administrative capability and per- 
formance. The management systems project is supported in part by an Advanced Insti- 
tutional Development Program (A! DP) grant to KVCC. The project is currently develop- 
ing several management systems to support the planning, management, and evaluation 
(PME) activities within the College. One component of the management systems project 
is called "outcome measures." This component wilt: (1 ) identify institutional outcomes 
and corresponding measures, (2) develop the instruments and procedures for data gather- 
ing, analysis, and reporting, and (3) provide guides to administrators for their use of re- 
ported outcome measures data in decision-making. 

The development of outcome measures brings to mind two anecdotes with which 
you may be familiar. It was reported that when Henry Ford was trying to develop safety 
glass for his automobiles, engineers on the staff weren't able to find a way to do so. He 
ditected his chief engineer to hire new engineers because they wouldn't knowthe ways 
how not to make safety glass. Of course, now all cars have safety glass. Then there is a 
story told about Tom Edison. After he had tried about a thousand different ways to 
make an incandescent light bulb, someone asked him if he were discouraged. He said, 
"No, I have just learned a thousand ways howm;/ to make a light bulb." Eventually he 
was successful, because we now have the incandescent light bulbs which he invented. A 
few people, like Ford's engineers, told us that we could not identify, measure, and evalu- 
ate outcomes of the College. And, like Edison, we have found a number of ways how 
not to do the job. The purpose of this article is to share with you some of the KVCC 
experiences, both successes and problems, in carrying out the project. Here is the real 
picture. Some things we think were done right. The report will also reveal some of our 
errors and problems. After describing our experiences, this article will outline a model 
other colleges could use to plan and implement such a project. 



NEED FORTHEPROJIXT 

McManis Associates, Inc., in its work with the two-year college AIDP Consortium, 
has developed a PME checklist. The checklist can be used by colleges in reviewing their 
PME capability Questions are presented in three sections: planning, management, and 
evaluation. The following questions, included under evaluation, are related to outcome 
measures: 

7. Is there a clearly defined list of institutional outcomes linked to the in- 
stitutional objectives? 
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3, Are there regular follow-up studies of graduates and early leavers? 

6. Is evaluation infornhition linked to the institutional MIS and routinely 
fed back to the institutional planners? 

Thus, the checklist suggests that a college's PME system should include some capability 
for measuring and evaluating ol xomes. 

There were both external and internal concerns which led KVCC to include an out 
comes approach in development of its PME capability. External concerns were related 
to requests received from: (1 ) the State Bureau of the Budget asking for ouput measures, 
and (2) the State Department of Education's emphasis on student performance objectives, 
especially for vocational/technical programs, requiring some kind of measurement of re- 
sults. There were some internal concerns that related to the need for a more systematic 
approach to managing the College and to assuring ourselves that we are doing what we 
think we ought to be doing. The President and deans found themselves making decisions 
on an ad hoc basis in critical areas such as budget preparation, enrollment predictions, 
program continuation, and staffing. Another concern was that administrators were evalu- 
ated annually by their supervisors and there' was not general agreement on the criteria for 
effective performance. We were concerned about the lack of a set of criteria for reviewing 
instructional programs that would . )ly over a period of time and that was generally 
known. Another concern was that udch office had its own data files, thus reports of data 
from one office did not always agree with reports from another. We were also concerned 
that each administrate^ and faculty member had his own interpretation of the catalogue 
statement of "purpoo. dnd objectives" to guide performance of his duties and responsi- 
bilities. 

It became obvious that we needed to be a little more systematic about administering 
the College. We were nof ineffective, nor were we inefficient. However, we saw the need 
to improve our performance by becoming more effective and more efficient. So we con- 
cluded that through some systematic approaches, we could increase our effectiveness and 
improve our efficiency. The comprehesive management systems project, of which out- 
come measures was a component, was planned and initiated to help further develop the 
effectiveness and efficiency of the College's PME capability. 



PROJECT PHASES 

The outcome measures project began at KVCC in January, 1974, and after two years 
is still Incomplete. A review of events and tasks completed to date revealed that the proj- 
ect will have passed through four phases during its life. The outcome measures project 
had a planning phase, a developing phase, a refining phase, and we will soon enter an im- 
plementing phase. Let us briefly examine each of those phases. 
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Planning Phase, The planning phase included the following activities. An outside 
consultant was used to assist KVCC in the project. The consultant interviewed all key 
administrators and used the information from those people about perceptions and ex- 
pectations of the project to develop a tentative model. At about this time, the need be- 
came evident for some type of documentation of the various management systems. The 
approach we chose was a management systems manual with a section for each system, in- 
cluding outcome measures. Our next step was to outline that section on outcome measures. 
We outlined an outcomes inventory by slightly modifying the structure developed by Dr. 
Sidney Micek at the National Center for Higher Education Management Systems (NCHEMS). 

We then examined a number of references we were able to discover. Outcome vari- 
ables was the first level of the inventory we developed. Those variables were then com- 
pared to the mission and goals statement. A questionnaire was drafted to determine more 
specifically what information administrators thought they needed to know to make deci- 
sions. This was comparable to Dr. Micek 's national "Outcome Measures Information 
Study." A progress report was used to conclude our planning phase. There was some 
overlap of tasks in the planning phase with those of the developing phase. 

Developing Phase. During the developing phase, we: (1) conducted the "need to 
know" study, (2) drafted the outcome measures inventory that was previously outlined, 
(3) drafted and re-drafted the outcome measures section of the management systems 
manual, and (4) identified some questions and issues related to how the project fit into 
the College operations. Further, we: (5) drafted some definitions of terms, (6) outlined 
an outcome measures taxonomy, and (7) gave progress reports to administrators. This 
led into the refining phase. 

Refining Phase. In the refining phase, we: (1) analyzed the discrepancies between 
outcomes data needed and college goals, (2) drafted a technical manual, (3) drafted a 
users' guide, and (4) designed a community impact study to examine some environmental 
factors which affect administration of the College. Future tasks planned for this phase 
include: (5) developing a dictionary of terms, (6) defining the roles of data managers 
(people who collect, store, retrieve and report outcome measures data within the insti- 
tution), (7) preparing illustrative reports of outcomes measures data as a trial implemen- 
tation, and (8) conducting a users' workshop to help administrators know limitations, ad- 
vantages, and possible uses of the data which will be reported to them. 

Implementing Phase. In this phase we will: (1 ) complete the outcome measures 
section in the management systems manual,* (2) begin to implement the technical pro- 
cedures, (3) complete the community impact study. (4) use MBO to integrate the uses 
of outcome measures into the PME, and (5) begin to evaluate the college outcomes. 



• In thai section, we will have a technical manual which is a set of data handling instructions to guide 
data managers and research and evaluation staff. The management systems manual will also include 
the users' guide which will be a reference for decision-makers who will be using the information pro- 
vided to them. Procedures for updating and augmenting the system and some reference material will 
also be included. An outline of the information included in the technical manual is contained hcrem 
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PROJEC1 PROBLFMS 



Some of ihe problems we encountered in the project resulted from our lack of ex- 
perience on which to base planning. Other problems occurred because we did not com- 
mit adequate staff time to carry out planned tasks. 

The outcome measures project was initiated before a statement of college mission 
and goals was available. Absence of such a statement delayed the project by nearly three 
months. 

We were not clear about the results expected from the project. Project objectives 
and tasks changed sevei :! times as we progressed through the planning phase. 

Communication among college staff, and with the consultant, was difficult and 
somewhat unproductive due to a lack of term definitions. Different people were using 
the same terms to mean different things and, in some cases, using different terms to mean 
the same thing. 

We greatly underestimated the amount of staff time the project would need. Our 
estimates would have been more accurute if the other problems had not occurred. How- 
ever, we had very limited reserve or contingency resources to commit to the project when 
it was found that additional staff would be needed. 

It was difficult for people not working directly with the project to see any value it 
could provide to them. Project staff shifted from telling others everything as it occurred 
(this was confusing to them) to reporting only concrete results (which were difficult to 
understand). There was need to: (1 ) focus on issues raised by the project, and (2) dis- 
seminate interim products which resulted from resolution of those issues. 

Dealing with these problems helped us become much wiser. Because of them, we 
now know some of the ways no/ to carry out such a project. We hope you can benefit 
from our experience. The experience has led to an outline of the process for developing 
an outcomes oriented PME. 

MODEL FOR OUTCOMES ORIIZNTED PME 

Any college which undertakes a project to improve its PME capability by measuring 
and evaluating its outcomes could adapt a model such as the one outlined on the follow- 
ing page. Experience with the KVCC project suggests a three-stage model which can be 
implemented by completing eleven steps. The three stages involve delineation, measure- 
ment, and evaluation of college outcomes. 
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STAGE!. DELINEATION OF OUTCOMES 



Step 1. Determine Benefits 
Step 2. Establish Goals 
Step 3. Delineate Outcomes 



STAGE 11. MEASUREMENT OF OUTCOMES 

Step 4. Select Measures 

Step 5. Collect Data 

Step 6. Analyze Data 

Step 7. Report Analyses 



STAGE HI. EVALUATION OF OUTCOMES 

Step 8. Compare to Plans 

Step 9. Evaluate Planning 

Step 10. Evaluate Management 

Step 1 1 . Evaluate Evaluation 



SUMMARY 

A number of external and internal concerns led the President and administrators at 
KVCC to recognize the need for use of outcome measures in planning, management, and 
evaluation of the College. The outcome measures project designed to respond to that need 
progressed through four phases in its development. The phases were: planning, develop- 
ing, refining, and implementing the project. Problems encountered by the project resulted 
from lack of experience on which to base planning and the lack of adequate staff time to 
carry out planned tasks. The experience suggested a three-stage, eleven-step model for a 
college to develop outcomes oriented PME. Appendix C contains the tentative list of the 
outcome measures developed at KVCC through use of this process. 
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ERIC 



AiM'lZNDIX A 



A LIST OF THE OUTCOME MEASURES IN THE 

i')7> ovicoMi: Mi ASVRi.s AM) rR()(i:nrRi s mam \ i 
I n i l) REMi w i.nmoN 



A. Stmlcnt Kiiowlotlge ;nul Skills Dcvclopnionl Oulconies 

A-1 Student development concerniny />m/f/r// of knowlinJyj^ 

A-2 Student development concerning depth of knowledge 

A-3 Student success in passing certificcition ^nd licensing exnmiiuuioris 

A-4 Areas and agents of student chonge during college 

li. Student I'ducatioiKil C;iavr DevelopmcMil Otilionies 

B-1 Highest degree or certificat(? planned 

B 2 Students enrolled in an orgafiizcd educational activity for no credit 

B 3 Program completers during a certain time period 

B-4 Program completers who entered as transfer students 

B-5 Degrees and certificates earned by an entering class of studeias 

B-6 Time to program completion for a ^radjiadnt: class 

B-7 Time to program completion for an entering class 

B-8 Educational program dropouts 

B-9 Students seeking additional degrees and certificates 

B-1 0 Students working toward and receiving another degree .^r certificate 

B-1 1 Student ability to transfer credits 

B-1 2 Level of achievement of forn jr students in another institution 

C. Student Ediicntionni Satisfaction Outconies 

C-1 Student satisfaction with overall educational experience 

C-2 Student satisfaction with vocational preparation 

C-3 Student satisfaction with knowledge and skills in the humanities 

C-4 Student satisfaction with critical thinking ability 

C-5 Student satisfaction with human relations skills 

n. Student Occupational Career Development Outcomes 

D-1 Student success in obtaining//rvr job 

D-2 Student success in obtaining /?rt'/VmY///r.s7 job 

D 3 Occupational career choice 

D-4 Job satisfaction 

D-5 First job earnings 
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Appondix A 
Pa()0 2 of 3 



1) (5 Annu.n tot.il incorni? of lt)rrni!r sludtMUs 
I) / l:in()k)vrn<?nl in r]i..iit)r lioklol study 
I.) B ChtWH}*- 'i'id sli»l)ililY ol (MftMM (joiHs 

Sliidnil I\ms(mi;iI I )i»vol()pninit 

(No outcome ineLisua'S and (kiUi acqnisii iofi ()ro(:t:dur(Js nre [)n)S(Mit(Hl in this cdli? 
(jory in this viMsion of th(^ nuinuiil.) 

I''. Sdidonl Soimm! ( nllniiil Dcvclopnionl 

(No outcome measures and data accjuisition [)ro(:«'(lur es jrc prcstMited in this cntc 
(joiy in this version Oi the? iTUiruiiil.) 

(i. I)cvcl()|)nicii! ol Now Know led i.'.e 

G 1 F^escLirch proposals funded 
G-2 Resocircfi restricted rt^venuos 

H. Dcvolopnicnt of New Ail I oinis 

(No measures and procedures art? pr(?sented in this category in this version of the 
manual.) 

I. COinninnitN Impact : I diication 

1-1 Enrollment of non-degree/diploma/certificate students 

1-2 Community participation in community education programs 

1-3 Community participation in extension services 

1-4 Educational goals achieved by community participants 

J. Coniniunity Impact: Service 

J-1 Institution's participation in community affairs 

J-2 Community participation in an institution's social, cultural, and recreational 
programs 

J-3 Community use of institutional facilities 

K. Coniniunity Impact: licononiic 

K-1 Institution's payment of local and state taxes and tax compensation 

K-2 Institution's purchase of locally provided utilities 

K-3 Institution's purchase of locally delivered goods and services 

K-4 Institution's capital equipment expenditure relevant to the local cornrnunity 

K-5 Institution's capital construction expenditure relevant to the local community 
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AfiponcliH A 
PiuH) 3 of 3 



K () Loc.il i?x|)tindiluM)s l)y Licuhy .uui st.ill 
K / Lociil (?x|)t)iuliUiM?s I)/ stiui(Mils 
K R Local (?xptMuliliii(V.; l)y visitois 
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AIMM NOIX n 
SAMPLl. ABSTRACT Sfn:l:T AND 

connrsi^ONDiiMG data acquisition iMK)C[:DURr s 

l-OR ONI. or J\\[': NCIUEMS OUTCOMf: MrASURLS 



NATIONAL (:r.NTt:n I C)I{ M(c;MCH riOUCAMON MANA(U:MrNT SYST[ VIS 



0 2 



MEASURE 
NUMOEM 



MEASURE NAME 



!>ti)(|fnt Mifcrs', It, {)\)\,\\n\\)i] prt'ft'f rt\/ f ir\t nth 



DEFINITION 

NnnilxM .jnd pi.'i ccntai)'' nt studt'[it\ wfio i('i;i'iv('d Ihc )oh of ftfst r.how.r upon 

t.'jvnM] Ttv institution 



OATASOURCES 

Toi rnt'f students 



PROCEDURES 

Administration of a snrvoy (^uestionnj'o.' 



COMMENTS 

Tfiis outcom*' mcasun? is KhMUdied as a poterTtitil nicusMrf,' of Vocational Prcpaniticti 
(1.4.1.02) in th(? NCHEMS I nvuntory of Hicjhi.'r Education OuTcome Varud^lfjs jnd 
Measures ■ s(M; A[:)f)enclix A 



Appofxlix n 
? of f) 



ACQUISITION P»U)CLDUIU S fOM OUICOMI Ml ASIMU D ? 



Tfn! protiiMluM? il(.'vrk)|)('cl toi ll^f' .it:t)U isit inn uf h)i DUtcoinc ijumsum' D ic 



PROCf DURE FOR A FORMER STUOrMT OUESTiONNAlRr 



It shoukl bii uoU)(\ thill {lUt.'Siioruuiih? ilfins 1 aiul 2 ofi ilu' fi)lU)wi(H) p.u)!.' jri.' sii{) 
(j(?slocj since they am conifKirHhle to ihos<3 used in thu Arnericun CvHincii on h ducjtion's 
lon()ituclinjl follow iip sludi<?s of coIIim]»; students. As .1 result, r(?sulis ol)t.jin(i(l trcnn 
the use ot lliese ilenis ci\n bi; ■ 'npcirod to ilie ACE f(?suk.^ relcMcrirod m hcvcfitiny, Col- 
li'sic Dropouis (Astin, 1975) <'i t^v wntinn Ak.xdfuier Asiiti. Cncuiujte Scliool ol 

Education, University ol Calilornui ;it |._os Anij^ntv;, Cjidornia. 
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1. In your first full-time job after leaving 



[Name of Instrtution) , which of ths following was your first 
employer? (PLEASE CHECK ONF) 

First Employer 



Government: 
Education: 



Federal 

State & Local ■ • 

Elementary & Secondary 
Higher Education ...... 



Other Non-profit 

Organizdtions Hospitals, clir^i^s 



Social welfare . 

Church 

Other non-p'^^^f'^ Of ya ni i^at I ons 



(01) 
(02) 

(03} 
(04) 



(05) 
(06) 
(07) 
(08) 



Business & 



Service: 


Solf-emploV»-<^^ f^'' f^'^nnily 












(09) 




Private company . . 




(10) 




Professional partner,^ . . . , 




(11) 








(12) 


Other: 














(13) 



2. How much of the ^vork this first full-time job was devoted 
to the followir^gactivitics^ 'CHECK ONE FOR EACH ACTIV- 





(1> 
A Major 
Amount 


(2) 
A Minor 
Amount 


(3) 
None 


A. Teaching 








B. Research and Development 








C. Administration or Management 








D. Service to patients or clients 








E. Other (Specify) 









3. Was your first full- ti me jot) after leaving [Name of Institution] 
the lob yf^i-J most preferred tht? ttme^ 

(1) Y(JS 

(2) No 



4. Which statement best describes how you regarded 
your /^/rsf job^ (PLEASE CH ECK ON E) 

( 1 ) Temporary job until a better one could 

be found. 

_ (2) Ter.iporary job while waiting to r(?port 
to a new job. 
(3) Temporary job to earn money to do some- 
thing ols(! (travel, school, havt? free time, 
etc.). 

(4) Job to earn money wtiile I df.>cid«; what 

kind of vvoi k I wanted. 

(5) Job "^'^Ih possible care^M r:>oteniial. 

(6) Job vvith definite career potential. 

(7) Other (Please sf)(^cify) 



5. How did you find your first job after leavuuj (Name 
of institution! ? (PLEASE CHECK ONE) 

(1 ) Employed at job while completing niy 

program. 

(2) School [)lacement of^:ccr oi iristructor. 

(3) Professional periodicals or organt/ations. 

(4) Civil Service application, 

(5) Public or private employment agency. 

(6) Newspaper advertisement. 

(7) Dii ect ap()hcation to employiir. 

(8) Friends or t elatives. 

(9) Other (Please specify) 

6. Was your first job related to your major field of 
study (your program)^ 

(1 ) Directly reiatecJ 

(2) Somewhat related 
(3) Not related at all 



EKLC 
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As an alternative to questionnaire items 1 and 2 on the preceding page, one may wish to 
to consider the following three questions which have been used by the U.S. Bureau of 
Census to determine the employment sector and kind of work a person is doing or has 
done. Responses to these questions and those pertaining to questions 4, 5, and 6 can 
be compared to data the Bureau of Census collects in its annual educational attainment 
survey which is conducted each March. (See U.S. Bureau of Census Ciirrvfit Population 
Reports: Series P-2'), "Educational Attainment in the United States.") 



1. In what kind of business or industry was (or is) your first full-time job after leaving 
[Name of Institution] ? (For example, accounting firm, public school, TV manufac- 
turer) 



2. What kind of work did (or does) your /"/rsr full-time job involvo? ( For example, ac- 
counting, teaching chemistry, electrical engineering) 



3. In your first job were you (or are you): 

(1 ) An employee of a private business? 

(2) Self-employed in your own business? 

(3) An employee in local, state, or Federal Government? 

(4) A public employee in a non-government organization? (For example, an 

environmental agency, a public school system, a public hospital) 
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Another alternative for questionnaire items 1 and 2 on the preceding page is the following: 



1. From the list that appears on page . which entry best describes your first full- 
time occupation after leaving [Name of Institution] ? 
(Please write its 3-digit code in the bpace below.) 



APPENDIX C 



TENTATIVE LIST OF OUTCOME MEASURES 
DEVELOPED BY KALAMAZOO VALLEY COMMUNITY COLLEGE 



1.00.00.00 STUDENT GROWTH AND DEVELOPMENT 
1.10.00.00 General Education and Knowledge 

.01 .00 Students Completing Requirements for General Pre-Baccaiaureate Programs 

.01 Associate in Arts and Associate in Applied Science degrees awarded each 
year by career center 

.02.00 Transfer of Credits to Other institutions 

.01 Transcripts of credits forwarded to other institutions each year 

.02 Students transferring from KVCC who have been offered scholarships, 
grants and awards each year 

.03.00 Participation in Non-credit Campus Educational Experiences 

.01 Students attending college sponsored cultural and recreational activities 
each term 

.02 Students attending college sponsored non-credit workshops, lectures and 
seminars each term 

.03 Students completing courses for non-credit (audit) by development cen- 
ter each term 

.04.00 Programs for Non-Traditional Students 

.01 Credits earned each term by students classified as "C" on admissions form 

.02 Credits earned each term in courses offered before 8:00 a.m., between 4:00 
and 7:00 p.m., and on weekends (Saturday and Sunday) 

.03 Persons over 60 years of age completing courses each term 

.05.00 Impact on KVCC Educational Experience on Student's Perception of Man 
and Society 

,01 Selected students' social, political and racial attitudes measured during first 
term of attendance at KVCC and at time of graduation 
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.06.00 Measures of Student Ability to Communicate Effectively 

.01 Graduating students scores on test indicating their ability to read 

.02 Graduating students scores on tests that indicate their ability to write 

.03 Graduating students scores on tests that indicate their ability to communi- 
cate orally 

1 .20.00.00 Specialized S':;ll and Kno w led^e 

.01 .00 Students Completing Requirements in Technical Education 

.01 Students successfully completing certificate programs each year by field of 
specialization 

.02 Students graduating from AAS degree programs each year by field of spe- 
cialization 

.03 Students passing certification or licensing exams each year 

.02.00 Employment of Graduates of Certificate and AAS Degree Programs 

.01 Graduates who are employed in their area of preparation within three 
months after completion of training 

.02 Students who take a first job outside of their area of specialization each 
year 

.03 Starting salaries of graduates of certificate and AAS degree program each 
year 

1.30. 00. 00 Satisfaction vv//// A' VCC Support A ctivities 

.01.00 Student Recommendations 

.01 Students who recommend attendance at KVCC to friends and relatives 
measured every two years 

.02 Students attending KVCC on the recommendation of a student or former 
student measured every two years 

.02.00 College Placement Service 

.01 Placement of students in off-campus jobs before graduation 

,02 Students registered with placement service each year 

.03 Job interviews scheduled for marketing graduates each year 
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.03.00 Impact of Student Aid 

.01 Graduates who received scholarships, grants or aid during last year of atten- 
dance at KVCC by amount of aid and source 

.02 Graduates who received scholarships, grants or aid during first year of atten- 
dance at KVCC by amount of aid and source 

.03 Funds given to students in financial aid and scholarships each year 

.04 Monies paid to students for part-time, on-campus employment each year 

2.00.00.00 FACULTY AND STAFF DEVELOPMENT 

2. 1 0.00.00 Work Satisfaction 

.01 .00 Working Conditions 

.01 Communications initiated each year stating unfair treatment or adverse 
working conditions by job classification 

.02 Registrations each year stating dissatisfaction with working conditions by 
job classification 

.03 In-house transfers each year by organizational unit 

.04 Changes in pay status made each year (other than normal contractional 
increasement) 

.05 Mandays lost each year to strikes and work stoppage 

.06 Participants in KVCC in-service training programs each year by job classifi- 
cation 

3.00.00.00 COMMUNITY DEVELOPMENT AND SERVICE 

3.1 0.00.00 Community /Education Development 

.01.00 Effect of Job-Related Educational Experiences on Non-classified KVCC 
Students 

.01 Non-classified students who receive promotions as result of job-related edu- 
cational experiences each year 

.02 Non^classified students who receive salary increases as a result of job-related 
educational experiences each year 

.03 Non-classified students who obtained new jobs as a result of job-related edu- 
cational experiences each year 
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3.20.00.00 Community Services 

.01.00 Institutional Resources Allocated to Connnnunity Services 

.01 Budget allocated to connnnunity services each year 

.02 Estinnated monetary value of connnnunity services rendered each year by 
type of service 

.02.00 Individuals Served by College 

.01 Persons attending extra-nnural, cultural and recreational activities each year 
who are not students or ennployees of the college 

.02 Connnnunity residents not associated with KVCC as students or ennployees 
who received services fronn the college each year by type of service 

3.30.00.00 Comtnunity Impact 

.01.00 Employment and Distribution of Students 

.01 Graduates employed in KVCC service area mi, ed every three years 

.02 Distribution of KVCC graduates by geographical areas measured every 
three years 

.03 Annual salary granted KVCC graduates by selected employers measured 
every three years 

.02.00 Economic Impact 

.01 Dollar amounts of goods and services purchased by KVCC from service 
area every year by type of purchase 

.02 Impact of KVCC on decisions to locate business within the service area 
measured every fifth year 

4.00.00.00 PRODUCTS DEVELOPMENT AND UTILIZATION 

4. 1 0.00.00 Educational Materials, Products, and Reports 

.01 .00 Publications by Faculty and Students 

.01 Publications by KVCC faculty by type of publication reported each year 

.02 Publications by students by type of publication reported 

.03 Publications of KVCC staff used in KVCC classes each year 

.02.00 Educational Products 

.01 Educational products (other than publications) developed by KVCC per- 
sonnel in use on campus each year by type of product 
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.02 Technical inventions, patents, technical inriprovements on existing machines 
and devices by KVCC faculty, staff, and students reported each year 

.03.00 Development and Research 

.01 Funds allocated to development and research each year 

.02 Developmental and/or research projects completed each year 

4.20.00.00 Management Information and Reports 

.01.00 Institutional Planning 

.01 Institutional objectives met each year 

.02 Functional level objectives met each year 

.03 Dropouts before any credits are earned each term 

.04 Students who do not reenter the institution after absence of one term 

4.30.00.00 I'uhlic Information Materials and Reports 

.01 .00 Employers Awareness of KVCC Job Applicants 

.01 Employers contacted for job openings each term 

.02 Employers requesting job applicants from KVCC graduates 

.02.00 General Public Awareness of KVCC Institutional Program and Services 

.01 News releases which are used by newspapers, radio and/or TV each month 

.02 News releases devoted to faculty and administrator accomplishments which 
are used by newspapers, radio and/or TV each term 

.03 News releases devoted to student accomplishments which are used by news- 
papers, radio, and/or TV each term 

.04 Individuals and groups visiting college campus each month 

.05 College publications (including brochures, catalogs, etc.) printed and dis- 
tributed each year 

.06 Individuals enrolled who learned something about the college or its programs 
via mass media every three years 

.03.00 Articulation with Community Institutions 

.01 Visits to area high schools by college personnel each year 

.02 College programs and presentations to community groups and institutions 
each year 

.03 Articulation meetings between KVCC staff and baccalaureate granting 
institutions each year 
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